Method, device, and medium for processing image

ABSTRACT

The present disclosure relates to a method, a device and a medium for making up a face. The method for making up the face of the present disclosure includes: obtaining a first face image; determining facial key-points by detecting the first face image; generating a second face image by applying makeup to a face in the first face image based on the facial key-points; determining a first face region by segmenting the first face image, wherein the first face region is a face region that is not shielded in the first face image; and generating a final face makeup image with makeup based on the first face region and the second face image.

CROSS-REFERENCE TO RELATED APPLICATION

This application is based on and claims priority under 35 U.S.C 119 toChinese Patent Application No. 201910918021.4, filed on Sep. 26, 2019,in the China National Intellectual Property Administration, thedisclosures of which are herein incorporated by reference in itsentirety.

FIELD

The present disclosure relates to the field of deep learning technology,and in particular, relates to a method, a device, and a medium formaking up a face.

BACKGROUND

In common short video applications or camera applications currently, itis a common makeup technology to apply makeup (such as lipstick, eyeshadow and blusher) to the face. This makeup technology is relativelymature and has been widely used in various scenarios.

However, the interference of shielding information cannot be avoided andthe makeup cannot be applied to the face image in the electronic devicein the above makeup technology which affects the user experience.

SUMMARY

The present disclosure provides a method, a device, and a medium formaking up a face.

According to embodiments of the present disclosure, a method for makingup a face is provided, including:

obtaining a first face image;

determining facial key-points by detecting the first face image;

generating a second face image by applying makeup to a face in the firstface image based on the facial key-points;

determining a first face region by segmenting the first face image,wherein the first face region is a face region that is not shielded inthe first face image; and

generating a third face image based on the first face region and thesecond face image.

According to embodiments of the present disclosure, an electronic deviceis provided, including:

a processor; and

a memory configured to store instructions executed by the processor;

wherein the processor is configured to execute the instructions toperform the method for making up the face according to the embodimentsof the present disclosure.

According to embodiments of the present disclosure, a non-transitorycomputer-readable storage medium is provided and configured to storeinstructions which are executed by a processor of an electronic deviceto enable the electronic device to perform the method for making up theface according to the embodiments of the present disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

The drawings herein are incorporated into and constitute a part of thedescription, show embodiments conforming to the present disclosure, andare used to explain the principles of the present disclosure togetherwith the description and do not constitute an undue limitation of thepresent disclosure.

FIG. 1 is a schematic flow diagram of a method for making up a faceshown according to an embodiment of the disclosure.

FIG. 2 is a schematic diagram of a face image shown according to anexemplary embodiment.

FIG. 3 is a schematic diagram of facial key-points shown according to anembodiment of the disclosure.

FIG. 4 is a schematic diagram of a fitted lip position shown accordingto an embodiment of the disclosure.

FIG. 5 is a schematic diagram of preliminary makeup shown according toan embodiment of the disclosure.

FIG. 6 is a schematic diagram of a semantic segmentation result of aface shown according to an embodiment of the disclosure.

FIG. 7 is a schematic diagram of final makeup shown according to anembodiment of the disclosure.

FIG. 8 is a schematic structural diagram of another device for making upa face shown according to an embodiment of the disclosure.

FIG. 9 is a schematic structural diagram of a terminal applying a methodfor making up a face shown according to an embodiment of the disclosure.

DETAILED DESCRIPTION OF THE EMBODIMENTS

To make those of ordinary skill in the art better understand thetechnical solutions of the present disclosure, the technical solutionsin the embodiments of the present disclosure will be described belowclearly and completely in conjunction with the accompanying drawings.

It should be noted that the terms “first”, “second” and the like in thedescription and claims of the present disclosure and the above-mentioneddrawings are used for distinguishing similar objects, and do not need tobe used for describing a specific order or sequence. It should beunderstood that data so used are interchangeable under appropriatecircumstances so that the embodiments of the present disclosuredescribed herein can be implemented in an order other than thoseillustrated or described herein. The implementation modes described inthe following embodiments do not represent all implementation modesconsistent with the present disclosure. Rather, they are merely examplesof devices and methods consistent with some aspects of the presentdisclosure as detailed in the appended claims.

In common short video applications or camera applications currently, itis a common technique to apply makeup (such as lipstick, eye shadow andblusher) to the face Most common makeup solutions at present involveusing key points of the face, such as key points in a lip region, to fita corresponding curve and render makeup. These solutions are relativelymature and have been widely used in various scenarios.

A method for making up a face according to the embodiment of the presentdisclosure as shown in FIG. 1 may be applied in an electronic device,such as a mobile phone, a tablet, a computer and a television. Thespecific technical solution of the method is performed as follows.

Step S1: obtaining a first face image.

Step S2: determining facial key-points by detecting the first faceimage.

Step S3: generating a second face image by applying makeup to a face inthe first face image based on the facial key-points.

In some embodiments, an image is acquired, face detection is performedon the image, and the image is expanded according to a certain ratiobased on a result of the face detection, to obtain a first face imagecontaining the face as shown in FIG. 2. Of course, if an image does notcontain a face, the image is not processed. Then, as shown in FIG. 3,facial key-point detection is performed on the first face imagecontaining the face to obtain facial key-points, and a makeup is addedto the first face image shown in FIG. 2 according to the facialkey-points shown in FIG. 4 and pre-set face makeup region information.The second face image is an makeup-applied face image is shown in FIG.5.

It should be noted that the preset part contains a face makeup mode andposition information of the face makeup region. The face makeup mode is,for example, applying lip gloss to the lips of the face, or renderingeye shadow on the eyes of the face. The position information of the facemakeup region is position information corresponding to the face makeupmode. For example, if a selected face makeup mode is applying lip glossto the lips of the face, then the position information of the facemakeup region is position information of key points of the lips.

Step S4: determining a first face region by segmenting the first faceimage, wherein the first face region is a face region that is notshielded in the first face image. In some embodiments, the first faceregion is determined by segmenting the first face image based on asemantic segmentation model, wherein the first face region is a regionin the first face image other than a shielded region and a backgroundregion.

In some embodiments, based on a semantic segmentation model obtained bytraining and the first face image, the semantic segmentation result,i.e., the first face region, as shown in FIG. 6, is the non-shieldedface region. A shielding object and a background are both classified asnon-face regions.

In some embodiments, the semantic segmentation model is obtained aftertraining by a deep neural network model through the following steps.

Each of a plurality of training sample images is used as an input, therespective one semantic segmentation result output corresponding to eachtraining sample image is compared with a labeled result of therespective one training sample image, and training is performed based ona difference between the semantic segmentation result outputcorresponding to each of the training sample images and the labeledresult of the respective one training sample image, so that thedifference between the semantic segmentation result output and thelabeled result meets a requirement, wherein the semantic segmentationresult corresponding to each training sample image is the non-shieldedface region in each training sample image, and the labeled result is asemantic segmentation result labeled in the training sample images inadvance. In the training to generate the semantic segmentation model, amachine learning algorithm, a convolutional neural network (CNN)algorithm or the like may be used, which is not limited in theembodiments of the present disclosure.

In some embodiments, the non-shielded face region is divided out by thedetected facial key-points. For example, when the detected facialkey-points in a left eye region of the face are less than apredetermined number, then the left eye region of the face is determinedto be shielded.

In some embodiments, a mask image of face shielding information, asshown in FIG. 6 is obtained by setting a pixel value of the non-shieldedface region in the semantic segmentation result to 1, and pixel valuesof other regions to 0.

Step S5: generating a third face image based on the first face regionand the second face image.

In some embodiments, a shielded region and a non-shielded region in amakeup region in the second face image is determined based on the firstface region; and the third face image is generated by remaining themakeup of the non-shielded region and removing the makeup of theshielded region.

In some embodiments, the makeup-applied face image as shown in FIG. 5 iscorrected according to the semantic segmentation result shown in FIG. 6.Specifically in the correction, a first overlapping region between themakeup region and the non-shielded face region, and a second overlappingregion between the makeup region and the shielded face region aredetermined, and then makeup in the second overlapping region in themakeup-applied face image is removed, while only retaining makeup in thefirst overlapping region in the makeup-applied face image, to obtain afinal face makeup image with makeup as shown in FIG. 7.

In some embodiments, the mask image of the face shielding information isused to retain the makeup in the first overlapping region and remove themakeup in the second overlapping region.

FIG. 8 is a block diagram of an electronic device 1300 shown accordingto the embodiments of the disclosure, including.

a processor 1310; and

a memory 1320 for storing instructions executable by the processor 1310,

wherein the processor 1310 is configured to execute the instructions toperform the method for making up the face in an embodiment of thepresent disclosure.

In some embodiments of the disclosure, the processor is configured toexecute the instructions to:

obtain a first face image;

determine facial key-points by detecting the first face image;

generate a second face image by applying makeup to a face in the firstface image based on the facial key-points;

determine a first face region by segmenting the first face image,wherein the first face region is a face region that is not shielded inthe first face image, and

generate a third face image based on the first face region and thesecond face image.

In some embodiments of the disclosure, the processor is furtherconfigured to execute the instructions to: determine a shielded regionand a non-shielded region in a makeup region in the second face imagebased on the first face region; and generate the third face image, byremaining the makeup of the non-shielded region and removing the makeupof the shielded region.

In some embodiments of the disclosure, the processor is furtherconfigured to execute the instructions to: determine a first overlappingregion where the makeup region overlaps with the first face region, andremaining the makeup of the first overlapping region; and determine asecond overlapping region where the makeup region does not overlap withthe first face region, and removing makeup in the second overlappingregion.

In some embodiments of the disclosure, the processor is furtherconfigured to execute the instructions to determine the first faceregion, by segmenting the first face image based on a semanticsegmentation model, wherein the first face region is a region in thefirst face image other than a shielded region and a background region.

In some embodiments of the disclosure, the semantic segmentation modelis trained based on a difference between a labeled result and a semanticsegmentation result; wherein the labeled result is a face regionpre-labeled in training sample images, and the semantic segmentationresult is a semantic segmentation result of segmenting each of thetraining sample images by the semantic segmentation model.

In some embodiments of the disclosure, the processor is furtherconfigured to execute the instructions to: obtain an image; determine aface region by recognizing a face in the image; and acquire the firstface image based on the face region, wherein the first face imagecomprises the face region, and the area of the first face image isgreater than that of the face region.

In some embodiments of the disclosure, a storage medium including one ormore instructions, such as a memory 1320 including instructions, is alsoprovided. The above-mentioned instructions may be executable by theprocessor 1310 of the device 1300 to perform the above-mentioned method.In some embodiments, the storage medium may be a non-transitorycomputer-readable storage medium. For example, the non-transitorycomputer-readable storage medium may be an ROM, a random access memory(RAM), a compactdisc read-only memory (CD-ROM), a magnetic tape, afloppy disk, an optical data storage device, or the like.

In an embodiment of the present disclosure, as shown in FIG. 9, theembodiment of the present disclosure provides a terminal 1400 applyingthe method for making up the face provided by an embodiment of thepresent disclosure, the terminal including the following components: aradio frequency (RF) circuit 1410, a power supply 1420, a processor1430, a memory 1440, an input unit 1450, a display unit 1460, a camera1470, a communication interface 1480, and a wireless fidelity (Wi-Fi)module 1490. Those skilled in the art may understand that the structureof the terminal shown in FIG. 9 is not a limitation on the terminal. Theterminal provided by the embodiment of the present application mayinclude more or less components than in the illustration, or involves acombination of some components, or a different component arrangement.

The components of the terminal 1400 are specifically introduced below inconjunction with FIG. 9.

The RF circuit 1410 may be used for data reception and transmissionduring communication or a call. Specifically, after receiving downlinkdata of a base station, the RF circuit 1410 transmits the data to theprocessor 1430 for processing, in addition, the circuit transmits uplinkdata to be transmitted to the base station. Generally, the RF circuit1410 includes, but is not limited to, an antenna, at least oneamplifier, a transceiver, a coupler, a low noise amplifier (LNA), aduplexer, or the like.

In addition, the RF circuit 1410 may also communicate with otherterminals through wireless communication and a network. The wirelesscommunication may use any communication standard or protocol, includingbut not limited to the global system of mobile communication (GSM),general packet radio service (GPRS), code division multiple access(CDMA), wideband code division multiple access (WCDMA), long termevolution (LTE), e-mail, short messaging service (SMS), etc.

The Wi-Fi technology is a short-distance wireless transmissiontechnology. The terminal 1400 may be connected to an access point (AP)through the Wi-Fi module 1490, so as to achieve access to a datanetwork. The Wi-Fi module 1490 may be used for receiving andtransmitting data during communication.

The terminal 1400 may be physically connected with other terminalsthrough the communication interface 1480. In some embodiments, thecommunication interface 1480 is connected with a communication interfaceof the other terminal through a cable to achieve data transmissionbetween the terminal 1400 and the other terminal.

In the embodiment of the present application, the terminal 1400 canimplement a communication service and send information to othercontacts, so the terminal 1400 needs to have a data transmissionfunction, that is, the terminal 1400 needs to include a communicationmodule therein. Although FIG. 9 shows communication modules such as theRF circuit 1410, the Wi-Fi module 1490, and the communication interface1480, it may be understood that in the terminal 1400, there is at leastone of the above-mentioned components or other communication module(such as a Bluetooth module) for implementing communication to performdata transmission. For example, in the case where the terminal 1400 is amobile phone, the terminal 1400 may include the RF circuit 1410, and mayalso include the Wi-Fi module 1490; in the case where the terminal 1400is a computer, the terminal 1400 may include the communication interface1480, and may also include the Wi-Fi module 1490; and in the case wherethe terminal 1400 is a tablet, the terminal 1400 may include the Wi-Fimodule.

The memory 1440 may be used for storing software programs and modules.The processor 1430 executes various functional applications and dataprocessing of the terminal 1400 by running the software programs andmodules stored in the memory 1440, and after the processor 1430 executesprogram codes in the memory 1440, part or all of the processes in FIGS.1 and 8 of embodiments of the present disclosure can be realized.

In some embodiments, the memory 1440 may mainly include a programstorage area and a data storage area. The program storage area may storean operating system, various application programs (such as acommunication application) and a face recognition module, etc. and thedata storage area may store data created based on the use of theterminal (such as various pictures, video files and other multimediafiles, and face information templates), etc.

In addition, the memory 1440 may include a high-speed random accessmemory, and may also include a non-volatile memory, such as at least onemagnetic disk storage device, flash memory device, or other volatilesolid-state storage devices.

The input unit 1450 may be used for receiving numerical or characterinformation input by a user, and producing a key signal input related touser settings and functional control of the terminal 1400.

In some embodiments, the input unit 1450 may include a touch panel 1451and other input terminals 1452.

The touch panel 1451, also known as a touch screen, may collect theuser's touch operation thereon or in the vicinity thereof (for example,the user's operation on the touch panel 1451 or in the vicinity of thetouch panel 1451 using any suitable object or accessory such as a fingeror a stylus), and drive a corresponding connection device according to apreset program. In some embodiments, the touch panel 1451 may includetwo parts: a touch detection device and a touch controller. The touchdetection device detects the user's touch orientation, and detects asignal brought by the touch operation, and transmits the signal to thetouch controller; and the touch controller receives touch informationfrom the touch detection device, converts the touch information intocontact coordinates, and then transmits the contact coordinates to theprocessor 1430, and can receive a command sent by the processor 1430 andexecute the command. In addition, the touch panel 1451 may beimplemented in various forms such as resistance, capacitance, infrared,and surface acoustic waves.

In some embodiments, the other input terminal 1452 may include, but isnot limited to one or more of a physical keyboard, a function key (suchas a volume control key, and a switch key), a trackball, a mouse, and ajoystick.

The display unit 1460 may be used for displaying information input bythe user or information provided for the user and various menus of theterminal 1400. The display unit 1460 is a display system of the terminal1400, and is used for presenting an interface to achieve human-machineinteraction.

The display unit 1460 may include a display panel 1461. In someembodiments, the display panel 1461 may be configured in the form of aliquid crystal display (LCD), an organic light-emitting diode (OLED), orthe like.

Further, the touch panel 1451 may cover the display panel 1461, and whenthe touch panel 1451 detects a touch operation thereon or in thevicinity thereof, a signal is transmitted to the processor 1430 todetermine the type of touch event, and subsequently, the processor 1430provides a corresponding visual output on the display panel 1461according to the type of touch event.

Although in FIG. 9, the touch panel 1451 and the display panel 1461 areimplemented as two independent components to achieve input and outputfunctions of the terminal 1400, in some embodiments, the touch panel1451 is integrated with the display panel 1461 to achieve the input andoutput functions of the terminal 1400.

The processor 1430 is a control center of the terminal 1400, isconnected with various components by using various interfaces and lines,and performs various functions and data processing of the terminal 1400by running or executing software programs and/or modules stored in thememory 1440 and calling data stored in the memory 1440, therebyimplementing various services based on the terminal.

In some embodiments, the processor 1430 may include one or moreprocessing units. In some embodiments, the processor 1430 may integratean application processor and a modem processor, wherein the applicationprocessor mainly processes the operating system, a user interface,application programs and the like, and the modem processor mainlyprocesses wireless communication. It may be understood that theabove-mentioned modem processor may also not be integrated into theprocessor 1430.

The camera 1470 is used for implementing a photographic function of theterminal 1400 to photograph pictures or videos. The camera 1470 may alsobe used for implementing a scanning function of the terminal 1400 toscan a scanned object (two-dimensional code/bar code).

The terminal 1400 further includes a power supply 1420 (such as abattery) for supplying power to various components. In some embodiments,the power supply 1420 may be logically connected to the processor 1430through a power management system, to achieve functions of managingcharging, discharging, power consumption and the like through the powermanagement system.

It should be noted that the processor 1430 in the embodiment of thepresent disclosure may perform the functions of the processor 1310 inFIG. 8, and the memory 1440 stores contents in the processor 1310.

In addition, in some embodiments of the disclosure, the presentdisclosure further provides a storage medium, configured to storeinstructions which are executed by the processor of the above-mentioneddevice for making up the face to enable the above-mentioned device formaking up the face to execute the method for making up the face in theembodiments of the present disclosure.

After considering the description and practicing the invention disclosedherein, those skilled in the art will readily conceive of otherembodiments of the present disclosure. The present application isintended to cover any variations, uses or adaptive changes of thepresent disclosure, and these variations, uses or adaptive changesfollow the general principles of the present disclosure and includecommon general knowledge or customary technical means in the technicalfield not disclosed in the present disclosure. The description andembodiments are regarded as exemplary only, and the true scope andspirit of the present disclosure are indicated by the following claims.

It should be understood that the present disclosure is not limited tothe precise structure already described above and shown in the drawings,and various modifications and changes can be made thereto withoutdeparting from the scope thereof. The scope of the present disclosure isdefined only by the appended claims.

What is claimed is:
 1. A method for processing an image, comprising:obtaining a first face image, determining facial key-points by detectingthe first face image; generating a second face image by applying makeupto a face in the first face image based on the facial key-points;determining a first face region by segmenting the first face image,wherein the first face region is a face region that is not shielded inthe first face image; and generating a third face image based on thefirst face region and the second face image.
 2. The method according toclaim 1, wherein said generating the third face image comprises:determining a shielded region and a non-shielded region in a makeupregion in the second face image based on the first face region; andgenerating the third face image, by remaining the makeup of thenon-shielded region and removing the makeup of the shielded region. 3.The method according to claim 2, further comprising: determining a firstoverlapping region where the makeup region overlaps with the first faceregion, and remaining the makeup of the first overlapping region; anddetermining a second overlapping region where the makeup region does notoverlap with the first face region, and removing makeup in the secondoverlapping region.
 4. The method according to claim 1, wherein saiddetermining the first face region comprises: determining the first faceregion by segmenting the first face image based on a semanticsegmentation model, wherein the first face region is a region in thefirst face image other than a shielded region and a background region.5. The method according to claim 4, wherein the semantic segmentationmodel is trained based on a difference between a labeled result and asemantic segmentation result; wherein the labeled result is a faceregion pre-labeled in training sample images, and the semanticsegmentation result is a semantic segmentation result of segmenting eachof the training sample images by the semantic segmentation model.
 6. Themethod according to claim 1, wherein said obtaining the first face imagecomprising: obtaining an image; determining a face region by recognizinga face in the image; and acquiring the first face image based on theface region, wherein the first face image comprises the face region, andthe area of the first face image is greater than that of the faceregion.
 7. An electronic device, comprising: a processor; and a memoryconfigured to store instructions executed by the processor, wherein theprocessor is configured to execute the instructions to: obtain a firstface image; determine facial key-points by detecting the first faceimage; generate a second face image by applying makeup to a face in thefirst face image based on the facial key-points; determine a first faceregion by segmenting the first face image, wherein the first face regionis a face region that is not shielded in the first face image; andgenerate a third face image based on the first face region and thesecond face image.
 8. The electronic device according to claim 7,wherein the processor is further configured to execute the instructionsto: determine a shielded region and a non-shielded region in a makeupregion in the second face image based on the first face region; andgenerate the third face image, by remaining the makeup of thenon-shielded region and removing the makeup of the shielded region. 9.The electronic device according to claim 8, wherein the processor isfurther configured to execute the instructions to: determine a firstoverlapping region where the makeup region overlaps with the first faceregion, and remaining the makeup of the first overlapping region; anddetermine a second overlapping region where the makeup region does notoverlap with the first face region, and removing makeup in the secondoverlapping region.
 10. The electronic device according to claim 7,wherein the processor is further configured to execute the instructionsto: determine the first face region, by segmenting the first face imagebased on a semantic segmentation model, wherein the first face region isa region in the first face image other than a shielded region and abackground region.
 11. The electronic device according to claim 10,wherein the semantic segmentation model is trained based on a differencebetween a labeled result and a semantic segmentation result; wherein thelabeled result is a face region pre-labeled in training sample images,and the semantic segmentation result is a semantic segmentation resultof segmenting each of the training sample images by the semanticsegmentation model.
 12. The electronic device according to claim 7,wherein the processor is further configured to execute the instructionsto: obtain an image; determine a face region by recognizing a face inthe image; and acquire the first face image based on the face region,wherein the first face image comprises the face region, and the area ofthe first face image is greater than that of the face region.
 13. Anon-transitory computer-readable storage medium, configured to storeinstructions which are executed by a processor of an electronic deviceto enable the electronic device to: obtain a first face image; determinefacial key-points by detecting the first face image; generate a secondface image by applying makeup to a face in the first face image based onthe facial key-points; determine a first face region by segmenting thefirst face image, wherein the first face region is a face region that isnot shielded in the first face image; and generate a third face imagebased on the first face region and the second face image.
 14. Thenon-transitory computer-readable storage medium according to claim 13,wherein the non-transitory computer-readable storage medium is furtherconfigured to enable the electronic device to: determine a shieldedregion and a non-shielded region in a makeup region in the second faceimage based on the first face region; and generate the third face image,by remaining the makeup of the non-shielded region and removing themakeup of the shielded region.
 15. The non-transitory computer-readablestorage medium according to claim 14, wherein the processor is furtherconfigured to execute the instructions to: determine a first overlappingregion where the makeup region overlaps with the first face region, andremaining the makeup of the first overlapping region; and determine asecond overlapping region where the makeup region does not overlap withthe first face region, and removing makeup in the second overlappingregion.
 16. The non-transitory computer-readable storage mediumaccording to claim 13, wherein the non-transitory computer-readablestorage medium is further configured to enable the electronic device to:determine the first face region, by segmenting the first face imagebased on a semantic segmentation model, wherein the first face region isa region in the first face image other than a shielded region and abackground region.
 17. The non-transitory computer-readable storagemedium according to claim 16, wherein the semantic segmentation model istrained based on a difference between a labeled result and a semanticsegmentation result: wherein the labeled result is a face regionpre-labeled in training sample images, and the semantic segmentationresult is a semantic segmentation result of segmenting each of thetraining sample images by the semantic segmentation model.
 18. Thenon-transitory computer-readable storage medium according to claim 13,wherein the non-transitory computer-readable storage medium is furtherconfigured to enable the electronic device to: obtain an image;determine a face region by recognizing a face in the image; and acquirethe first face image based on the face region, wherein the first faceimage comprises the face region, and the area of the first face image isgreater than that of the face region.